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Abstract 

Researchers at Harvard University, World Education, Education Development Center, and in Bolivia 
recently concluded a large-scale longitudinal study over three years in Bolivia to clarify whether a 
relationship similar to that found between girls’ education and development exists between the basic adult 
education for women and women’s social and economic well-being and development. The study 
randomly sampled 2, 200 adult women with 1,600 participants in the programs and 600 non-participants. 
The researchers comprehensively measured the construct of women’s social and economic well-being and 
development using indicators, including: 1) literacy / education level; 2) participation in income-earning 
activities; 3) household decision making; 4) children’s education; 5) civic participation and awareness; 6) 
legal rights participation and awareness; and 7) health care participation and awareness. A key research 
finding of the study was that the integrated basic adult education programs have a significant program 
impact (or effect) on Bolivian women’s lives even after taking into consideration the effects of women’s 
individual characteristics, formal education level, socio-economic status, home location, and the unknown 
factors during the period of 3 years. The participants experienced a net gain of 1 1 percentage points on 
the index of social and economic well-being over the non-participants. The paper explains the research 
framework applied, methods used, and key findings emerged. 
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1. Introduction 

It is quite evident in literature that girls’ schooling leads to a positive social change by “imparting skills 
and fostering other individual changes that alter women’s patterns of social participation” and their 
economic life (Levine, Levine, & Schnell 2001). But little research has been carried out on the effects of 
programs aimed at improving the literacy and basic social and economic skills of women. In particular, 
studies that attempt to measure the magnitude of the changes in women’s social and economic 
participation or the extent to which programs are affected by factors such as inputs (financial resources, 
facilitators, and training time or focus), intake (women’s characteristics), and women’s household socio- 
economic status are especially rare. In this paper, we will present findings that illustrate not only the 
magnitude of the effects of integrated literacy and basic education programs on women’s social and 
economic well-being, but also identify factors that help to explain the effect and effect size. Additionally, 
we will present the results of a cost-effectiveness analysis of these programs. 

This paper discusses findings from a recent longitudinal survey, of 2,200^ women in Bolivia carried out 
during 1998-2000, under the Girls’ and Women’s Education Policy Research Activity (GWE-PRA), 
funded by the US Agency for International Development, Office for Women and Development. This 
research was part of a GWE-PRA multi-country initiative. Longitudinal studies were conducted in three 
countries: Bolivia, Honduras, and Nepal. The Bolivia study assessed the impact of women's integrated 
literacy and basic education programs in four NGOs (PLAN, Gregoria Apaza, Pro Mujer, CRECER) and 
one NGO partnership (PLAN/CRECER) on women's social and economic well-being and development 
by studying specific social and economic indicators. These included: 1) literacy and education; 2) 
children's education; 3) health and reproductive health; 4) participation in economic activities; 5) 
household decision making; 6) community participation; and 7) awareness of legal rights. In addition, 
technical assistance was provided to participating NGOs in the areas of research design, materials 
development, and monitoring and evaluation. 

2. Research Context 

The longitudinal study in Bolivia began in 1998 when the country was suffering from an economic 
downturn. During the period that the research was underway, earnings were down and many people lost 
their jobs. According to the World Bank (2003), in 1999, growth experienced in the preceding years 
slowed “in part due to tight government budget policies, which limited needed appropriations for anti- 
poverty programs, and the fallout from the Asian financial crisis.” In 2000 overall growth was limited to 
2.5% largely because of major civil disturbances, as well as a result of the global slowdown and “laggard 
domestic activity” (World Bank, 2003). 

In 1998, policy questions regarding adult education programs for women were raised among international 
donor communities. Should the policy makers in donor communities continue to invest limited resources 
in adult education programs for women when many young girls are not enrolled in schools? Do these 
adult education programs have any significant social and economic impact on the participants? Are these 
programs cost effective? Obviously, these are not easy questions but prompted a launch of this 
longitudinal study in response to the debate as to whether support should continue to be provided for 
integrated literacy and basic education programs for women, when resources are insufficient. This policy- 
driven study was, therefore, undertaken for the purpose of identifying the extent to which integrated and 
adult basic education programs have an effect on women’s social and economic well-being. 



^ Of the 2,200 women in the original sample, 941 were interviewed in all three years. 
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3. Research Design 

The research design for the Bolivia study is a comprehensive quasi-experimental design with a 
longitudinal tracking of the same sample cohorts over three years (1998-2000)^ It also consists of both 
qualitative and quantitative data collection. A total of 2,200 women were randomly selected in Year 1 
with 1,600 from an experimental group and 600 from a control group. The experimental group of the 
program participants and the control group of non-participants, located in the departments of La Paz and 
Cochabamba, were all in the age cohorts between 15 and 45. None of the selected women, of the 
experimental or control group, had participated in the programs prior to the study. The sampled women 
were tracked for three years while the experimental group participated in the programs and the control 
group did not. By Year 3, the total sample size was down to 941 women with 717 in the experimental 
group and 224 in the control group. 

Inherent in the research design was the assumption that changes in the participants' knowledge, attitudes 
and practices on indicators of social and economic well-being would ultimately result in changes in the 
larger society. The research focused on the immediate impact on women within the three-year period of 
the study. These data were examined in conjunction with information concerning program elements, such 
as facilitator characteristics and program length and focus. 

3.1 Conceptual Model 

In order to visually illustrate the comprehensive design, we developed a conceptual model to capture 
essence of the research hypothesis. See Diagram 1 : Conceptual Model of the Bolivia Research Design . 
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Figure 1: Conceptual Model of the Bolivia Research Design 



^ The design was “quasi-experimental” because participation in the intervention or experimental group (integrated literacy and 
basic education programs) was voluntary and the participants’ selection process was not controlled. However the longitudinal 
data allows us to monitor how the baseline difference between the experimental and the control groups change over time. 
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As depicted in Diagram 1, input information was provided on seven indicators of social and economic 
development, as well as on their background and overall socio-economic status at baseline. Meantime, 
input also indicates that the education programs started with participants enrolled and educational 
resources provided. The process level represents women's participation in the programs within the study 
time frame. Outcome level data show changes in the knowledge, awareness, attitudes, and skills/practices 
that may be attributed to the programs, and thus, provide an estimate of the program's impact on their 
social economic development over a three-year period. The hypothesis was clear that all participants in 
the programs would have more social and economic gains measured by seven areas of indicators than the 
non-participants over time. 

Additionally, a sub-sample of 30 women from the experimental group and five focus groups (two from 
Pro Mujer, one from Gregoria Apaza, one from PLAN/CRECER, and one from PLAN) was selected and 
interviewed to provide in-depth information about the women, their families, and the communities in 
which they resided. The selection process started in year 2 and the criteria of the selection were based on 
the quantitative analysis of the baseline year (1998). Women with the highest and lowest levels of social 
and economic indicators were picked for the qualitative interviews. 

It is important to note that none of the participating NGO programs were traditional literacy training 
programs, but rather socially and economically focused training programs, such as a health and 
reproductive health program, a micro-credit and microenterprise program, a technical skills program, and 
so forth. Furthermore, participants in these programs were not "typical" literacy program participants- 
they began the program with higher levels of literacy skills than one would find in a traditional literacy 
program. 

3.2 Research Questions 

Across the span of three years (1998-2000), we expected that many positive changes would occur in 
women’s awareness of and behavior in education, health, economic participation, legal right issues, and 
empowerment. The major challenge was that we must find out how much the integrated literacy and basic 
education programs for women would contribute toward additional improvements in their personal well- 
being (private returns). Three key research questions what guide the overall research are: 

1. Does women’s participation in integrated basic education programs lead to improvements in their 
social and economic well-being? 

2. To what extent do the programs contribute to improvements in women’s social and economic well- 
being after other factors are taken into consideration? 

3. Are the integrated literacy and basic education programs for women cost-effective? (Are the programs 
worth the money invested?) 

These are not easy research questions, given the abstract nature of the key outcome measure-“social and 
economic well-being.” In order to precisely answer these research questions and closely measure the 
construct of social and economic well-being, we used a matrix to develop a 154-item survey instrument to 
assess women’s knowledge, attitudes, and behavior of the seven areas of social and economic indicators. 
The data instrument would allow us to devise a single but comprehensive index measuring women's 
social and economic well-being. This index formed the basis for the analysis of the multivariate effects of 
the five programs, along with alternative contributing factors. 
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3.3 Index of Women’s Social and Economic Participation 

In this multi-year study, the development of a valid and reliable scale of women’s social and economic 
well-being was essential to assessing the programs’ salient influences and relationships that improve 
social and economic conditions for women. Although the concept of creating an index was simple and 
was carefully planned in the early design stage of the project, the challenge of constructing a sensitive and 
comprehensive index for individual social and economic well-being was enormous. The vision of creating 
an index that is appropriate measure of women’s social and economic status in Bolivia preceded the 
development of the survey instrument. This allowed us to craft every substantive question item in the 
instrument as contributing part of the overall index. 

The final index'^ consisted of multiple composites of key areas of indicators, such as literacy education, 
family and reproductive health, income-earning activity, household decision making, community 
participation and legal rights (Home SES, which measures woman’s household material asset, was not 
included in the index). This index was intended to mark the overall social and economic development of 
the group. A high score on the index means high social and economic well-being and a low score 
indicates low social and economic well-being. 
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Figure 2: Overall Measure to Indicate Women’s Social and EconomWell Being 



The index of the social and economic well-being, which is statistically reliable, served as the outcome 
variable in the multivariate analysis. The final statistical model used in this research led to the answers to 
our research questions, which we will describe in this paper. 



^ The index is constructed with 56 non- weighted variables related to women's knowledge, attitudes and behavior 
with respect to health, reproductive health, literacy and education, income-earning activities, legal rights, decision 
making, and community participation. The index score is normally distributed with a skewness statistics at -0. 1 1 . 
The estimated alpha of reliability is 0.9 1 . 
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4. Research Findings 

4.1 Women’s Social and Economic Well-Being by Group and Location 

In order to see how Bolivian women “scored” in this newly developed index of social and economic well- 
being, we examined the following statistics in Table 1, which shows the index statistics in the 
experimental and control groups by locations and the NGO programs over three years. 

Table 1: Overall Index Score of Women’s Social and Economic Well-Being 
by Group and Location Over Three Years 



Overall Index Score” Overall Gain Score 





Yearl 


Year 2 


Year 3 


Year 1-2 


Year 2-3 


Year 1-3 : 


By Group: 














Control 


22.9 


24.3 


25.4 


1.4 


1.1 


2.5 


Rural 


16.2 


15.8 


19.1 


-0.4 


3.3 


2.9 


Urban 


27.9 


30.6 


30.1 


2.7 


-0.5 


2.3 


Experimental 


28.6 


32.0 


32.4 


3.4 


0.4 


3.8 


Rural'’ 


22.9 


25.2 


27.0 


2.3 


1.8 


4.1 


Urban*’ 


32.5 


36.6 


36.2 


4.1 


-0.5 


3.6 


By NGO: 














Gregoria Apaza 


30.8 


34.2 


34.3 


3.4 


0.1 


3.5 


Pro Mujer 


33.8 


38.8 


37.0 


5.0 


-1.8 


3.2 


PLAN 

International 


18.3 


18.0 


20.8 


-0.3 


2.8 


2.6 


PLAN/CRECER 


22.5 


24.7 


26.1 


2.2 


1.4 


3.6 


CRECER 


31.0 


36.0 


37.4 


5.0 


1.4 


6.4 



“Based on a maximum of 56 items. 

'^Includes PLAN/CRECER and PLAN International participants, all of whom are in rural areas, and those of the CRECER 
participants who are in rural areas. 

^ Includes Gregoria Apaza and Pro Mujer participants, all of whom are in urban areas, and those of the CRECER participants 
who are in urban and semi -urban areas. 

In year 1 (the baseline year), the experimental group was almost 6 points higher on the overall index than 
the control group, with scores of 28.6 and 22.9 points for the experimental and control group, 
respectively. This gap widened in year 2, reflecting a larger gain for the experimental group (3.4 points) 
than for the control group (1.4 points), although both groups made some progress in social and economic 
well-being. But in year 3, progress was much slower for both groups, with a gain of 1.1 point by the 
control group and a gain of 0.4 points by the experimental group. In short, by year 3, the gap between the 
experimental and the control group on the index remained large, at 7 points. 
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In addition, large differences existed in the index between rural and urban areas. In all three years, rural 
women had a much lower score than urban women. In year 3, for example, the difference between 
women in rural and urban areas in the control group was more than 1 1 points. In the experimental group, 
the difference was more than 9 points. Specifically, women in rural NGOs (PLAN and PLAN/CRECER) 
had the lowest social and economic well-being index scores in years 1-3. PLAN also had the lowest 
overall gain score over time (2.6 points). CRECER showed the highest overall index score increase (6.4 
points). 

4.2 Women’s Social and Economic Well-Being by Household SES 

For the analysis of the impact of SES on women's well-being (private returns) at baseline (year 1), we 
divided women in the sample into four groups on the basis of their relative household SES ranking. Thus, 
we grouped women by SES into four quartiles, using a proxy measure of SES-household material 
possessions composite (0-14 points). The lowest quartile of SES consisted of 468 women (127 from the 
control group and 341 from the experimental group), with SES scores less than 5 points. For comparison 
purposes, we also computed the index score for the highest quartile group-women who scored 9 or more. 
A total of 447 women (109 from the control group and 338 from the experimental group) belonged to the 
highest quartile group. Table 2 shows how these two groups (the lowest and highest quartiles of the SES 
proxy) improved from years 1 to 3, on the index of social and economic well-being. 

Women in the lowest SES group had a larger gain in social and economic well-being index scores than 
women in the highest SES group over three years. Within the lowest or the highest quartiles, women in 
the experimental group had a larger gain than women in the control group. The larger gains made over 
time by the women in the lowest quartile suggest that the most disadvantaged women were able to “catch 
up” socially and economically over time. The difference between participants and non-participants in the 
integrated literacy and basic education programs also demonstrated that these programs helped to improve 
private return to women, regardless of their household SES levels. 

Table 2: Overall Index Score Differences between the Lowest and Highest SES Quartiie 
_ . Over a 3-Year Period 



OveraU Index Score (56) Overall Gain Score 





Yrl 


Yr2 


Yr3 


Yr 1-2 


Yr 2-3 


Yr 1-3 


Lowest Quartile 














Control 

(n=92) 


13.0 


15.5 


18.4 


2.6 


2.9 


5.5 


Experimental 

(n=152) 


15.4 


21.1 


22.4 


5.7 


1.3 


7.0 


Highest Quartile 


Control 

(n=39) 


34.8 


34.7 


33.6 


-0.1 


-1.2 


-1.2 


Experimental 

(n=212) 


37.7 


39.5 


39.4 


1.8 


-0.2 


1.6 



Women’s SES and participation in the integrated literacy and basic education programs may be inter- 
related, resulting in covariate contribution to the improvement in women’s social and economic well- 
being index. Although we cannot disentangle the effects of SES from those of the integrated literacy and 
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basic education programs, a multivariate analysis in the section that follows better elucidates the 
relationship between these two factors. 

4.3 Women’s Social and Economic Well-Being by Education Level 

An important question in our research was whether location (urban and rural) or formal education level 
were significant factors associated with the improvement of social and economic well-being. To answer 
this question, we compared two sets of means.^ In both rural and urban areas, we witnessed improvement 
on the social and economic well-being index. By the third year (2000), the index score for the women in 
rural areas improved from 22.8 to 26.8 points, an 1 8% increase. For the women in urban areas, the index 
score improved from 33.3 to 36.7 points, a 10% increase. It is interesting to note that most of the gains 
for the rural women occurred between years 2 and 3. But all the gains for the urban women occurred 
between years 1 and 2. Additional research is needed to determine whether this could be a delayed effect 
for the rural women. 

Table 3 shows that women experienced an improvement on this index regardless of their educational 
level. The general trend seemed to be that women with no or lower education made larger gains than 
women with higher education. Women with no education may benefit the most from the integrated 
literacy and basic education programs. The results suggest that while the gains for the more educated 
groups tapered off between years 2 and 3, the gains for the non-educated women and women with lower 
levels of education continued to be in evidence. 



Table 3: Overall Index Score and Gains by Location and Educational Level 





Overall Index Score(S6) 


Overall Gain Score 


i 


Year 1 


Year 2 


Years 


Year 1-2 Year 2-3 Year 1-3 | 


Locality: 














Rural 


21.2 


22.9 


25.0 


1.6 


2.2 


3.8 


Urban 


31.4 


35.2 


34.8 


3.8 


-0.5 


3.3 


Education Level: 














No Education 


18.9 


22.1 


23.1 


3.3 


0.9 


4.2 


Primary Ed. 


27.0 


29.4 


30.2 


2.4 


0.9 


3.2 


Secondary Ed. 


30.7 


34.0 


34.3 


3.3 


0.3 


3.6 


Post-secondary Ed. 


37.3 


38.8 


39.5 


1.6 


0.7 


2.3 



All findings shown in above three tables are average indicators by categories. This simple bi-variate 
analyses allowed us to determine whether a significant difference exists between the experimental and 
control group, among between education levels, and between urban and rural areas. However, such 
analyses did not inform us as to whether variables were inter- or intra-related in explaining the variability 
in women’s social and economic well-being. To examine the inter- and intra-relationships of multiple 
factors associated with women’s social and economic well-being over time, we developed an 
“explanatory” model. 



^ A t-test was conducted to compare composite means in urban and rural areas and an ANOVA test was calculated 
for education levels. 
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4.4 What Factors Contribute to Women’s Social and Economic Well-Being? 

The aim of developing a critical explanatory model was to find out whether the improvement in women’s 
social and economic well-being over three years could be attributable to the integrated literacy and basic 
education programs after we partial out the known effects of various factors (independent variables). This 
requires a multivariate analysis with the index of social and economic well-being as the dependent 
variable. Thus, a linear regression function was used in the following explanatory model: 

S=a+biXi+b2X2+b3X3+b4X4+b5X5+b6X6+e 



S, Index of social and economic well-being 

а, Intercept (or constant) 

b„, Coefficients (or slopes) associated with X„ 

Xn, Factors (or independent variables) that may explain a portion of the variance in S 
e, A leftover portion of the variance in S that can not be explained by all Xs in the model. 

We selected six key factors for the regression model: 

1 . Group (experimental and control), 

2. Education level, 

3. Marital status, 

4. Location (rural and urban), 

5. Home SES score, and 

б. Time/years. 

The most critical factor for this model is the group variable, which indicates the effect of the combined 
integrated literacy and basic education programs in the context of several other factors that also affect the 
outcome. 

Statistical results of the final model are shown in Table 4, which illustrates that all factors inserted in the 
model significantly contributed to the social and economic well-being. We explain the details below. 



Table 4: Factors that Explain the Overall Index of Social and Economic Well-Being 





slope 

coefficient 


s.e. 


t- 

statistics 


p-value 


Constant value (or intercept) 


3.88 


0.51 


7.56 


0.000 


Group (experimental=l and control=0) 


5.94 


0.31 


19.08 


0.000 


Women’s educational leveF 


3.08 


0.21 


14.95 


0.000 


Marital status (yes=l, no=0) 


3.40 


0.31 


10.87 


0.000 


Locality (urban=l, rural=0) 


4.76 


0.32 


14.74 


0.000 


Home material possession score (0-14) 


1.49 


0.06 


24.29 


0.000 


Time/years (0=baseline, l=year2; 2=year3)*’ 


1.19 


0.16 


7.33 


0.000 


R-square 


0.55 








F-statistics 


574.51 








p-value (model) 


0.00 









“ Women’s education level is categorized into four levels, 0 = no education, 1 = primary school level (grades 1-5), 
2 = secondary level (6-12), and 3 = education beyond grade 12. 

** Time/years is treated as a continuous variable and used to calculate average growth on an annual basis. 
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In the above explanatory model,® 55% of the total variance (R-square statistics = 0.55) in the index of 
social and economic well-being is “explained” by the six factors in the model. Each factor in the model 
significantly explains a portion of variance in the index, controlling for other factors (all p-values = 0). 
Results from this model suggest that, within three years (1998-2000), women who participated in the 
integrated literacy and basic education programs made significantly more progress on the index of social 
and economic well-being than women who did not participate in these programs, even when we 
controlled for other significant factors (effects) such as location, educational level, household material 
possession index, marital status, and time/year.’ 

On average, considering all other significant factors, we can expect that a woman who participates in one 
of the NGO programs will gain 1 1 percentage points more than a woman who does not participate. The 
net effect of the integrated literacy and basic education programs is highlighted in Figure 3. 
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Figure 3: Net Effect of Integrated Literacy and Basic Education Programs on Women Social and Economic 

Well-being between 1998 and 2001 After Considering Other Factors Equal 



Using our standardized index of women’s social and economic well-being, we identified a “typical” 
woman from the sample as a comparison basis to illustrate the key finding. This woman, who is at the 
22-percentage point mark (i.e., she scored 12.5 points on a 56-point scale) is single, has received a few 
years of primary education, and is living in rural Bolivia. Her household socio-economic status 
(measured at 4 out of 14 points) is poor, and she has not participated in any integrated literacy and basic 
education program. During the three-year study, she would be expected to move upward by 4.3 



^ We also examined interaction between “group” and “education level and interaction term between “group” and “locality.” 
However, the “interaction effect” associated with these variables did not significantly explain the “left-over” variance in the 
overall composite score (associated p-values for two coefficients are 0.77 and 0.67 respectively). 

’ It should be noted that year 1 to year 3 gains were often larger for the composite measures than for individual variables. 
Program impact was much more evident when variables were examined in combination with each other, rather than individually. 
This is likely because of the interaction between multiple variables measuring a single construct. 
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percentage points on the scale of social and economic well-being because of unknown factors. A married 
woman, on average, is estimated to gain another 6 percentage points on the scale. If a woman lives in an 
urban area in Bolivia, she would add another 8.5 percentage points. With a secondary education level, 
she would add 5.5 more percentage points. Compared to a woman with a low socio-economic status (4 of 
14 points), a woman with a high socio-economic status (10 of 14 points) would have additional 16 
percentage points. After we take into consideration the differential effects of various factors, it is evident 
that the net effect of the integrated literacy or basic education program is that a woman who participates 
in one of these programs, on average, obtains 11 percentage points higher on the scale of social and 
economic well-being than a woman who does not participate. 

In addition, we estimated that, on average, US $26 were spent on each woman for an average of 30 hours 
of training spread out over three years in Bolivia. It is remarkable that such a small cost could bring 
about an 1 1 -percentage point improvement (the effectiveness measure) in a woman’s social and economic 
well-being within three years. 

In brief, much of the research presented here highlights the benefits of the integrated and basic education 
programs for women in Bolivia. We concluded that investments by development agencies, donors, NGOs, 
and governments in education for women in a poor country will significantly improve women’s social 
and economic well-being and, therefore, the country’s social and economic development. 

Two additional points need to be mentioned. First, the gap in social and economic development between 
rural and urban women remained large even when we controlled for variables, such as educational level, 
household possessions, participation in the integrated literacy and basic education programs and other 
factors. Women in rural areas were expected to score 8.5 percentage points lower than the women in 
urban areas over time. Second, marriage seemed to be an important factor in improving women's social 
and economic well-being. The difference between married and single women in the index was almost 6 
percentage points after we controlled for other significant effects, such as participation in integrated 
literacy and basic education programs, location (rural and urban) women’s educational level, household 
possession, and time. 

4.5 What Elements of the Integrated Literacy and Basic Education Programs Made the 
Difference In Improving Women’s Social and Economic Well-Being? 

In addition to the NGOs’ strong commitment to the integrated literacy and basic education programs for 
women and hard work in the often regarded as extremely difficult and impoverished environment in 
Bolivia, three major factors contributed to the significant impact of these programs on women’s social 
and economic well-being and development. These were: 1) program length and focus; 2) facilitators, and 
3) costs. 

4.5.1 Program Length and Focus 

Prior studies in other countries (such as Nepal) have demonstrated that literacy alone is insufficient to 
have a major impact on social and economic development. An integrated approach that combines literacy 
instruction with substantive content that is relevant to participants' daily lives has proven to be a more 
effective means of achieving such gains. 

In this study, we found that the focus of the integrated literacy and basic education program's content was 
a factor affecting participants’ performance on various indicators of development. The figure 4 shows 
that there is a clear relationship between the focus of the program content and shares of the participants' 
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performance on the social and economic measures*. If a program spent longer time and more effort on 
particular area of indicators, the size of that area gain on the index of the social and economic well-being 
is likely to be larger. For example, the biggest 3 areas of concentration of all the NGOs’ program are on 
the income-generating, health, and community participation, it is therefore that the shares of the same 3 
components in the measure of the improvement in social and economic well-being are also the largest. 
This study concludes that although all training programs have an positive effect on women’s participation 
in social and economic development, more focus on specific area of social and economic development in 
training lead to more improvement in that particular area. Training in income-earning activities and 
health-related issues are especially evident in Bolivia 
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Figure 4: 


Shares of Contribution by Various Indicators Are Related to the Length 
and Focus of the Training Programs 



4.5.2 Program Facilitators 

As expected, we found that facilitators of the training programs played an important role in the quality of 
integrated literacy and basic education programs. In Bolivia, we found that female facilitators were more 
effective than male facilitators in bringing about the desired training outcomes, even when we controlled 
for other factors such as age, income, education level, and facilitator socio-economic status. On average, 
women who were taught by female facilitators scored almost 1 .4 points higher on the index than women 
who were taught by male facilitators. This suggests that employing female facilitators could improve 
program success if all other variables remain the same. Why are female facilitators more effective than 
male facilitators in Bolivia? This may require further investigation. However, our qualitative data 



* To calculate the shares of contribution by various indicators in the index, we analyzed each of the six areas of indicators as 
individual outcome variable against all the same predicting factors. In other words, we ran six linear regression models with six 
areas of indicators as outcome variables one by one. By doing this we were able to calculate each proportion of the estimated area 
gain to the estimated gain (effect) of the overall index. 
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suggested that women participants in the programs regarded female facilitators as a role models. It is a 
common expectation that a facilitator’s job is taken by a man. If a woman does it, the participants seemed 
to be more impressed by a female facilitator than by a male facilitator. There seems to be a resistance 
against male facilitators among the women participants in Bolivia. One participant said “men are allowed 
in everything, but that is not the case for women. I want a female facilitator to teach us. . (Arminda, El 
Alto). Another participant chimed in “In my case, my father doesn’t want me to study, only my mother 
supports me. I see my facilitator (a woman) is better who support my mother’s idea.” (Maria, El Alto) 

It is surprising, however, that facilitators’ level of education was negatively correlated with program 
outcome after we controlled for other factors. Findings on the measure of facilitators' qualifications 
indicated that high participants' index scores on the social and economic indicators tended to be 
associated with facilitators with less education. One possible explanation might be the mobility and job 
opportunities available to facilitators who have higher level of education. Facilitators with high education 
levels might not stay in the programs because they are offered better paid positions elsewhere. 
Facilitators with lower level of education might be more persistent and willing to work with women in a 
more consistent manner, earning their trust and cooperation. Facilitators’ age, income and opinions about 
the program did not have a large impact on the outcome of the integrated literacy and basic education 
programs.® 



Table 5 Facilitator’s Overall Socio-Economic Index 



1 


slope 




t-staiistics 


p-value \ 


1 


coefficient 








Constant value (or intercept) 


28.81 


1.71 


16.87 


0.00 


Facilitator’s gender (0=male, l=female) 


1.37 


0.40 


3.46 


0.00 


Facilitator’s age 


0.10 


0.03 


3.25 


0.00 


Facilitator’s monthly income 


0.00 


0.00 


1.61 


0.11 


Facilitator’s opinion (0-42) 


-0.17 


0.04 


-4.76 


0.00 


Facilitator’s education level" (1-7) 


-2.55 


0.25 


-10.12 


0.00 


Women’s locality (Urban=l, Rural=0) 


4.59 


0.43 


10.76 


0.00 


Women’s home material possession score (0-14) 


1.72 


0.07 


23.81 


0.00 


Women’s time/years (0=baseline, l=year 2; 


1.42 


0.19 


7.46 


0.00 


2=year 3) 










R-square 


0.517 








F-statistics 


0.537 








p-value (model) 


0.000 









^Facilitator’s Education level: 1 = No education, 2 = Primary school, 3 = Middle school, 4 = High school, 5 = Technical studies, 
6 = Teacher’s College, 7 = University 

It is also worth noting that the facilitator’s income does not appear to have a significant effect on the 
outcome. The income level for these facilitators was quite low and did not vary much. In technical terms, 
the variability needed in this variable did not have enough statistical power to explain the variability in 
the index. A sensitivity analysis we conducted also informed us that it didn’t make a difference in the 
overall statistics of the model whether we inserted the income variable or removed it from the regression 
model. 



® Their slope coefficients are small although they are all significant. 
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5. Cost Effectiveness 

Policy makers often rely on cost-effectiveness analysis to determine program costs in relation to program 
effectiveness. This research has already provided evidence that participation in the integrated literacy and 
basic education programs had a significant impact on selected indicators of women's social and economic 
well-being in Bolivia over a three-year period. But, are the educational programs or program 
achievements worth the money invested? This is not a question that can be easily answered. However, it 
is challenge that needs to be tackled. 

First, it is necessary to define the terms, “costs” and “effectiveness” used in this study. In addition, we 
will explain how we developed the criteria used and the compromises made in materializing the concept 
of cost effectiveness. For example, it is important to note that the outcome "effectiveness" (value) in a 
cost-effectiveness analysis is limited. Associating the limited “values” with fixed costs, aggregated or 
disaggregated, is bound to lead to a constrained cost-effectiveness ratio. 

Let us start with the term “effectiveness.” Effectiveness is often, if not always, underestimated. 
Quantitative estimates of perceived effectiveness such as the effect of the integrated literacy and basic 
education programs on social and economic development do not include non-quantitative effects such as 
increased interest in learning; the cascade effect of knowledge on family members, fiiends and co- 
workers; and long-term effects on children once they become adults. The value of being educated 
contributes positively to social capital but can hardly be assessed in a single study with limited time and 
resources'®. Having stated the limitations of the term "effectiveness," we define it in this study as 
estimated gain in values of social and economic development indicators realized by women because they 
participated in the education programs. 

We divided costs into two major categories: recurrent and initial capital costs. In this study, we examined 
annual costs associated with various training programs in five NGOs (PRO Mujer, Gregoria Apaza, 
PLAN, CRECER, and the PLAN/CRECER partnership. Some standardization to calculate relevant costs 
was necessary. For example, costs among the five NGOs varied in many aspects: 1) items listed in the 
accounting books, as well as proportional allocation of funds to each item; 2) length of training programs 
(ranging from a few weeks to a few years, often with different spacing of classes); 3) content (ranging 
from micro credits to health issues and legal rights against domestic violence); 4) location (ranging from 
remote rural villages to urban centers); and 5) funding sources and associated constraints. 

To make cost data comparable, we estimated training unit cost per hour for each participating woman, 
based on recurrent costs. We used the total recurrent training costs divided by the total number of 
training hours and then divided by the total number of program participants. 

It is important to note that the NGOs did not all define “recurrent” or “initial” costs in the same way. For 
example, material (textbook) printing was listed as an “initial” cost item in one of the NGOs (this NGO 
prints out materials for multiple-year use), but as a “recurrent” cost in another. We also observed that the 
training of coordinators or facilitators was regarded as “initial” cost in one NGO but as “recurrent” cost 
item in another. These inconsistencies made standardization difficult and we compromised all final 
estimates in one way or another. However, we standardized all items under the recurrent costs category 
across five NGOs with regard to the following factors: 

1 . Facilitators 



'® For further understanding the concept of social capital, readers can review Professor Francis Fukuyama’s recent publications. 
His book “The Great Disruption: Human Nature and the Reconstitution of Social Order” 1999 is highly recommended. 
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• Salaries 

• Per Diem 

• Transportation 

2. Supervisors and managers 

• Salaries 

• Transportation 

• Support staff 

3. Training facilities 

• Insurance 

• Maintenance 

• Rental space 

• Utilities 

4. Supplies 

• Paper and pens 

• Printing materials production 

• Other basic supplies (wall posters, lamps, incentives. . .) 

To calculate the per hour cost of training each participant, we asked the NGOs to provide the total number 
of hours of training for each training cycle. 
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Information about the focus and cost of each NGO program is depicted on Table 6. 



Table 6: NGO Training Programs Focus and Costs 



Pro iVIujer 



Gregoria 



PLAN 



CRECER 



PLAN/ 

CRECER 



Content 

Income-earning 

Health 

Reading & writing 
Community participation 
Legal rights 
Child’s education 
Empowerment/DM 
Others (sewage/water) 


60% 

15% 

0% 

10% 

0% 

5% 

10% 

0% 


18% 

0% 

6% 

28% 

24% 

0% 

24% 

0% 


10% 

15% 

5% 

5% 

5% 

35% 

0% 

25% 


30% 

40% 

20% 

0% 

0% 

0% 

10% 

0% 


20% 

25% 

5% 

5% 

5% 

25% 

0% 

15% 


Average Recurrent Cost: 












Per hour/per trainee (US$) 


$0.59 


$0.66 


$ 1.05 


$0.62 


$0.84 


Index Baseline Score 












(Yr l)Overall Average Score 


30.79 


33.82 


18.29 


31.00 


22.52 


Income-generating 


1.99 


4.69 


1.46 


4.08 


3.40 


Health" 


12.44 


11.57 


5.08 


10.65 


5.22 


Reading & writing 


5.60 


4.81 


2.96 


4.08 


3.68 


Community participation 


2.07 


2.92 


2.04 


3.33 


2.87 


Legal rights 


5.54 


5.56 


3.35 


4.92 


3.60 


Empowerment/DM 


3.14 


4.26 


3.40 


3.96 


3.75 


Estimated Gain Index Score 












(NGO Coefficients) 


8.65"* 


4.28*** 


0.88 


10.02*** 


5.95*** 


Income-earning 

Health" 


2.64*** 

2.13*** 


0.29** 

- . ^*** 
1.49 


0.18 

-0.71* 


2.17*** 

3.38*** 


1.78*** 

0.46 


Reading & writing 
Community participation 


1.05*** 

1.50*** 


*** 

1.08 

0.60 


0.54*** 

0.74*** 


0.98*** 

2.09*** 


1.04*** 

1.76*** 


Legal rights 
Empowerment/DM 


0.63*** 

0.70*** 


0.57*** 

0.25** 


-0.15 

0.29** 


0.66*** 

0.75*** 


0.19 

0.74*** 


Cost-Effectiveness Ratio 
(Index Score/Unit Cost) 

Income-generating 

Health 

Reading & writing 
Community participation 
Legal rights 
Empowerment/DM 


14.66 

2.68 

0.54 

0.25 

0.12 


6.48 

0.08 

0.10 

0.03 

0.21 

0.09 


0.84 

0.02 

-0.10 

0.03 

0.04 

-0.01 


16.16 

1.05 

2.18 

0.32 

0.12 


7.08 

0.42 

0.14 

0.06 

0.10 

0.01 



* p-value <0.05; ** p-value <0.01; *** p- value <0.001 



® Final indicator, “health,” consists of three composites (21 items): 1) health practices, 2) knowledge of birth-control 
methods, and 3) knowledge of STDs. Children’s health was not included in the composite because not all women 
had children. Each other indicator consists of a 7-item composite. 
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It should be noted that these figures represent planned, rather than actual hours. However, according to 
NGO staff, there was little difference between the number of hours planned and those actually carried out. 
Additionally, the total number of participants (used as the denominator for calculating per unit per person 
cost) included all women who participated in the programs. No information was available about the 
number of women who dropped out of the program or when they dropped out. Information on Table 6 is 
conveyed in five sections: 

Content. The first part of the table depicts the estimated percentage of time allocated to each content 
area. We obtained this information by asking NGO representatives to estimate the percent of time the 
organization spent on training in each area. Each percentage serves as a proxy for measuring the 
percentage of total training resources committed to each subject area by NGO. 

Average Recurrent Cost. The second part of the table displays estimated average recurrent cost per 
program participant on an hourly basis. This is calculated by dividing annual program recurrent costs by 
the total number of training hours provided by each NGO. On the basis of this information, we estimated 
that it would cost between US$ 0.59 and 1.05 per hour to train each woman in one of the five NGO 
programs or partnerships. The programs usually grouped between 10 and 20 women together in a given 
training session. It is important to note that we did not include initial startup cost in this table and that 
these costs could differ substantially from NGO to NGO. 

Index Baseline Scores. The third portion of the table shows social and economic well-being Baseline 
Index Scores. These values represent average scores on the 56-item composite measure at baseline 
(1998), when the research started. This information also allowed us to measure the gain score in a 
meaningful way. With the exception of children’s education, the indicators listed under the overall index 
score were all significant contributors to the index. 

Estimated Gain Index Score. The fourth part of the table includes estimated index gain scores for social 
and economic development over three years. The estimated gain scores served as a measure of the 
“amount of effectiveness.” This estimated gain score (regression coefficient) is more accurate than the 
calculation of “net gain” score (indicator difference between years 1 and 3). Estimated gain takes into 
consideration factors such as baseline, location, marital status, education level, and household SES 
measure. 

Cost-Effectiveness Ratio. The last part of the table lists the calculated ratio of “amount of effectiveness” 
divided by unit cost as an indicator of cost effectiveness. It is important to emphasize that the 
methodology used to calculate this “cost-effectiveness ratio” is different from “cost-benefit” or “cost- 
efficiency” analysis. Additionally, please note that attaching a dollar value or its equivalent to 
effectiveness or finding ways to minimize costs while maintaining the achievement goals was not the 
purpose of this study. 

Examining the information on this table, we see in the first section that Pro Mujer showed the highest 
commitment to training in income-earning activities or small business enterprise (60%), while PLAN 
demonstrated the lowest commitment of training time to the subject area (10%). However, Pro Mujer 



Children’s education is not included in the index because all questions were asked only of a subgroup of women 
who had children in school. 

Estimated index gain score comes from regression analyses results in which each NGO (coded as a dichotomous 
variable) was inserted as a predictor variable together with locality, educational level, marital status, SES, and 
time/years. Each NGO coefficient (a slope coefficient) therefore indicates an estimated additional gain score added 
to the index if women are participating in that NGO program. 
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only focuses 5% of its program on training related to improving children’s education, while PLAN 
focuses 35% of its program on that topic. Each of these NGOs has a very different program focus. As a 
result, NGOs’ allocation of financial resources to each training topic can vary tremendously. 

It is important to explore the interrelationships between the different categories of data shown. For 
example, consider the link between the amount of time that NGOs allocate to a given subject and 
participants’ improvement on indicators related to those substantive areas. As shown on Table 6, 
CRECER had the highest percentage of time committed to health-related training among all NGOs 
(40%). Women in CRECER also had the largest net gain (3.38 points) on the health and reproductive 
health composite. This represents a 32% increase over the 10.65 composite score at baseline (also the 
largest among all NGOs). Similarly, Pro Mujer, which allocates 60% of its training resources to income- 
earning activities (the largest percentage among all NGOs) gained 2.64 points on the estimated gain 
index''' in the income-earning activities composite. This represents a 75% increase over the composite 
score in the baseline year (1.99). 

In other areas, such as legal rights, there is not enough evidence to support the relationship between 
program focus and program effect. However, bear in mind that the only NGO allocating a substantial 
time to activity related to legal rights was Gregoria Apaza (24%), which showed only minimal increases 
in that area (0.57 points). 

Based on these data, we can estimate which NGO programs are the most cost effective, with respect to the 
indicators measured in the GWE-PRA study. The ratio of the “estimated index gain score” in social and 
economic development to the average recurrent participant cost per hour was calculated to measure cost 
effectiveness. The scores had no benchmarks to compare against, but comparisons were made across 
NGOs. High ratios, relative to other NGO scores were an indication of "good" cost effectiveness on the 
variables measured in the study. 

We found that CRECER and Pro Mujer had the highest ratios (16.16 and 14.66 respectively). For each 
dollar spent per hour by CRECER on an adult woman, we can expect a 16.16 point gain on our index of 
social and economic development, even controlling for location, educational level, marital status, SES 
levels, and time. Similarly, the expected increase in the index for Pro Mujer is 14.66 points. For 
Gregoria Apaza and the joint program PLAN/CRECER, we would expect increases of 6.48 and 7.08 
points, respectively. For PLAN, we would expect to see an increase of 0.63 points. These figures may 
provide an indicator of the relative ranking among the NGOs and the NGO partnership in the study on 
their cost effectiveness on our indicators or social and economic development. However, caution must be 
exercised in drawing this conclusion. It may simply be an indication of which organizations have the 
closest fit between our indicators and the areas covered by their programs. 

For example, we see that the expected increase on our study indicators for each dollar spent for PLAN 
was only 0.63. However, this does not necessarily mean that PLAN activities are the least cost effective. 
These results only indicate that for each dollar (per participant) invested in the program, PLAN 
participants made the least progress on the indicators measured in the study. However, it does not tell us 
anything about the cost effectiveness of the areas included in PLAN'S program that are not included 
among the indicators used to calculate the composite variable. It should be noted that only 40% of 
PLAN'S program activities are focused on areas included in the composite index. About 25% of PLAN’S 
work is focused on helping local communities to manage safe drinking water and clean sewage/waste, and 



'‘‘All estimated gains were calculated after taking into account other important factors such as geographic location, 
marital status, household SES proxy, formal education level and time. 
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another 35% of the time is allocated to programs related to increasing parents’ involvement in children’s 
education, which are not part of the standardized component of estimated effectiveness. 

Additionally, we must recognize that there is great variation among the NGO programs in coverage area, 
class size, development of curriculum, and budget management (e.g. ratio of initial one time cost to 
recurrent cost) all of which affect cost effectiveness. These factors cannot be standardized and therefore 
cannot be built into our cost-effectiveness model. For example, PLAN works with women in remote rural 
areas where transportation, logistical planning and incentive programs could cost more. Class size could 
also be a sensitive variable to cost (and effectiveness). Usually, in rural areas, class size is smaller 
because communities needing access the programs are small and scattered. Additionally, all the non- 
standardized variations mentioned and their sensitive relationships to the cost were loosely (or never) 
reflected in our final analysis of cost effectiveness. 

6. Summary 

Overall, these findings indicated that the NGO programs examined did have a significant impact on 
women’s social and economic well-being. Moreover, we concluded that women who participated in the 
integrated literacy and basic education programs in Bolivia showed a larger improvement during the 
period than women who did not participate in these programs. Hence, we concluded that funding for 
these types of programs should be continued. More specifically, when the composite index was 
examined, we found that, even when taking into consideration other factors, such as education level, 
marital status, locality, home material possessions and time/year, the estimated average difference 
between the experimental and the control group in the composite index of social and economic 
development is 1 1 percentage points (5.5 percentage points annually). By Year 2 (1999), a woman who 
participated in one of the integrated literacy and basic education programs would obtain 5.5 percentage 
points more, on average, in social and economic well-being than a women who did not participate in a 
program. By Year 3 (2000), the same woman would continue to obtain an additional 5.5 percentage 
points, totaling 1 1 percentage points higher than non-participants in two years. 

Additionally, other factors, such as level of formal education, marital status, location (urban and rural), 
household socio-economic status and time/year were found to have a significant impact on women’s 
social and economic well-being. For example, for each additional level of education, a woman obtains 
(primary, secondary, and post-secondary), she can expect to score, on average, 5.5 percentage points 
higher in the index of social and economic well-being than less educated women. Married women also 
made greater gains than unmarried women on the index of social and economic development. Time was 
also a factor affecting outcomes. 

Findings revealed that areas of indicators in which NGOs have made substantial investment of resources 
(time, program focus, and training) have resulted in significant gains in women’s social and economic 
development. Therefore, it is essential for governments, donors, and NGOs to prioritize their goals and 
target resources toward achieving these goals. 

In examining factors contributing to the 1 1 percentage point gain in social and economic development 
(the pieces of the “pie”), we found that certain areas of indicators were not sufficiently supported, and 
hence, did not significantly contribute toward development gains. Priority should be given to selecting 
areas that deserve greater attention in the next phase of program development. For example, 
consideration should be given to investing additional resources in areas such as legal rights, decision- 
making, and children’s education. 

Involvement of in-country researchers, NGO staff, and other stakeholders in every aspect of policy 
research from initial design to the final analysis is essential to local capacity building, as well as 
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addressing relevant research issues and producing studies that more likely to be used by policy makers 
and program planners. 

The use of locally developed materials that are relevant to the local context (such as those developed by 
the four NGOs) is an important ingredient of effective program delivery. 

The study shows that, given the total cost of the programs examined in our study and considering the total 
number of women who participated in and benefited by an 1 1 percentage point gain on the measure of 
their social and economic well-being, the integrated literacy and basic education programs for women are 
cost-effective. It is estimated that it costs $17 dollars to train 20 women (an average class size in a 
village) for one hour in Bolivia. This includes facilitator salary, per diem, basic material costs, basic 
transportation, management and support staff, as well as maintenance and utility costs. 

Lawrence Summers, the current president of Harvard University, pointed to statistical evidence that 
“female education is the variable most highly correlated with improvements in social indicators” 
(Summers, quoted in Wigg, 1994). An 1 1 percentage points gain on the index of women’s social and 
economic well-being suggests that women are significantly more likely to: 

• become small entrepreneurs earning their own income, 

• take a sick person for medical treatment, 

• use a family planning method, 

• be involved in their children’s education, 

• make independent decisions, and 

• stand up against domestic violence 

After all, they are more empowered, more independent, more knowledgeable, more socially and 
economic able beings in society. 
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